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(57) Abstract 

In a data communication system 
having network traffic with Mows, a net- 
work traffic director system using router 
modules. The router modules direct traf- 
fic based on optimizing a merit function or 
penalty function to reduce costs of conges- 
tion for stochastically changing demands 
and Hows in the data communication sys- 
tem. The router modules exchange values 
with neighboring router modules. Based 
on the exchanged values and values lo- 
cal to a router module, flow conditions 
are checked and if necessary the local val- 
ues are adjusted until the flow conditions 
are satisfied or a time period expires. Ad- 
justments are associated with optimizing a 
merit function or penalty function. Based 
on the adjusted values, the router mod- 
ule adjusts parameters to be used to di- 
rect packets of the network traffic Mows 
to other router modules or other destina- 
tions within the data communication sys- 
tem. An aggregation scheme is used for 
reducing the number of values stored in a 
single router module. 
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METHOD AND APPARATUS FOR NETWORK CONTROL 



TECHNICAL FIELD 

The present invention is related generally to data communication 
systems and. in particular, to network control of data communication systems. 

5 BACKGROUND OI TI IE INVENTION 

Data communication systems generally consist of multiple networks that 
are linked together with bridges, switches, routers, or other devices. These linking 
devices direct data packets along various paths from an origination to a destination. 
Since a data communication system tends to be large with many networks and 

1 0 interconnecting links, typically a data packet can lake many paths from its origination to 
its destination. A goal of a network traffic director system is to direct the journey of 
each data packet from its origination to its destination so that the capacity of the 

network is efficiently utilized. 

Traditional network traffic director systems utilize a plurality of network 

1? routers based on Internet Protocol (IP). The IP routing approaches are based on a 
Shortest Path (SP) method in which packets are sent to their destinations along the 
shortest paths in the network. There are various methods to determine shortest paths 
among the IP routing approaches. However, these approaches all share a common 
assumption that ifeach packet travels along the shortest path, then the overall workload 

20 required for the routing will remain as small as possible. 

The assumption of traditional routing systems would hold true if the 
overall performance of a routing system could be adequately measured solely on the 
basis of the total number of decisions that routers need to perform. However, other 
factors are involved in the overall performance of a routing system. One of these 

25 factors is the amount of congestion in a data communication system. Some theoretical 
proposals exist which attempt to address the congestion of network traffic director 
systems. These approaches, however, are impractical in providing adequate solutions to 
the congestion problem. The end result of some of these approaches produces, on 
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average, flows that are little different than the shortest path methods of IP routing 
taking congestion into account. Another theoretical approach to congestion is intended 
for environments having predefined aspects, such as loading and network topology. 
However, a data communication system is seldom sufficiently predictable for these 
5 predefined approaches. 

Other theoretical approaches attempt to address an aspect of the 
stochastic (random) nature of flows in a data communication system by allowing for 
randomness in the arrival times of individual data packets into a linking device. These 
approaches, however, are still formulated in a deterministic manner by requiring that 
10 demand or load rates and (low rates in other parts of the communication system be 
known in advance of their actual occurrence. 

These prior art deterministic approaches are not ideally suited for data 
communication system environments in which demands or loads and flows in other 
portions of the data communication system have stochastic distributions due to 
15 changing demands or loads of individual users or due to any multiplexing involved. 
This means thai not only arc the arrival times of individual packets random, but also the 
rates of the demands or loads and flows in other portions of the data communication 
system are random and not known before their actual occurrence. 

An effective optimization method must not only minimize congestion in 
20 the data communication system but it must be robust enough to adapt to sudden changes 
in conditions of the data communication system such as packet arrival, loading on the 
system, and system topology without predefined scenarios of when or how these 
changes will occur. 

In data communication systems, loading changes quickly and 
25 dramatically due to such influences as changing demands or loads of individual users 
and multiplexing methods such as time division multiplexing (TDM) where time slots 
of system use are divided among individual users. With TDN4, a data communication 
system constantly changes the particular users on the system at any given moment, thus 
loading changes rapidly. Network topology can also have rapid stochastic changes due 
30 to situations such as equipment failures or weather conditions. 
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Also, whereas the prior art systems and methods focus on a single 
indicator- such as shortest path or average delay, as a basis to optimize data flow, there 
are many additional factors. These include average queue length at various data 
communication system linking devices, variance or standard deviation in individual 
5 delays of data packets traveling from an origination to a destination, and average and 
variance of the individual utilized capacity of links and linking devices in a data 
communication system. Thus, a system and method is greatly desired which optimizes 
packet How in a data communication system based on numerous factors while being 
robust enough to effectively adapt to unexpected events. 

10 SI 1MMARY Of I HI- INVHNTION 

The present invention is directed to a network traffic director system in a 
data communication system having network traffic with randomly distributed demands 
and Hows. In one aspect of the present invention the network traffic director system 
includes router modules configured to direct data packets in the tlata communication 

15 system, each router module being a home router module including a neighborhood 
.supervisor The neighborhood supervisor is configured to send home potentials of the 
home router module to neighborhood supervisors of neighboring router modules and to 
receive neighbor potentials of the neighboring router modules from neighborhood 
supervisors of neighboring router modules. The home router module further includes a 

20 dynamic load balancer configured to determine Hows based on the home and neighbor 
potentials. The dynamic load balancer adjusts the home potentials if first conditions 
including How conditions are not met. The dynamic load balancer also updates routing 
tables if second conditions based on the adjusted home potentials are met. The home 
router module also includes a dynamic data How splitter configured to receive data 

25 packets from networks and router modules. The dynamic data flow splitter selects a 
portion of the data communication system for each data packet received based on the 
updated routing tables. Each received data packet is transmitted to the portion of the 
data communication system selected by the dynamic data flow splitter for the received 
data packet. 
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Another aspect of the invention includes the home potentials of each 
home router module being associated with a pair of nodes of the data communication 
system including an origination node and a destination node. The home potentials of 
each router module arc also associated with a quality of service level. The 
5 neighborhood supervisor is configured to send the home potentials to neighborhood 
supervisors of neighboring router modules when the flow conditions arc not met. The 
How conditions include conservation of flows. The dynamic load balancer is 
configured to determine flows by using a response function associated with optimizing 
at least one of a penalty function or a merit function for stochastic demands or flows of 

10 the data communication system. The dynamic data flow splitter is configured to select 
the portion of the data communication system for each packet received further based on 
either a Markov protocol, a Routing Wheel protocol, previously selected portions of the 
data communication when an address for a received data packet is a same type as an 
address stored in the dynamic data flow splitter, or on a quality of service level 

1 5 associated with a data packet. 

In a further aspect of the invention the neighborhood potentials are 
stored according to an aggregation scheme. The merit or penalty functions are at least 
locally approximated by a quadratic function of the Hows. The ideal data flows are 
further based on resistance of arcs associated with the router module wherein each 

20 resistance of an arc is configured to be a function of cither arc capacity or a function of 
flows of the arc. The neighbor router modules are configured to be associated with the 
home director module through at least one of a political, organizational, topological, 
topical, or geographical relation. In a further aspect each router module is configured to 
adjust home potentials to an equilibrium point for a difference function associated with 

25 flows of arcs associated with the router module. The difference function is associated 
with differences between home potentials and neighbor potentials. The merit function 
or penally function involves costs of congestion including at least one of traffic delays, 
high latency, diminished throughput, lost packets or unresponsiveness to sudden 
changes in topology or loading. 



BNSOOCtD: <WO 00241 64A1 I > 



WO 00/24164 



5 



PCT/US99/24145 



BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a topology diagram representative of a network traffic 
director system of the present invention. 

Figure 2 is a block diagram showing interconnection detail of router 
5 modules of the network traffic director system of Figure 1 . 

Figure 3 is a block diagram showing internal detail of a router module of 

Figure 2. 

Figure 4 is a topology diagram showing internal detail of a router 
module of Figure 2. 

]0 Figure 5 is a relational diagram of an aggregation scheme used in the 

network traffic director system of Figure I . 

Figure 6 is a topology diagram showing internal detail of an arc of a 

router module of Figure 2. 

Figure 7 is a flowchart of a method used by the router modules of 

1 5 Figure 2. 

DETAILED DESCRIPTION OF THE INVENTION 

Large communication data systems are comprised of a collection of 
networks linked together by linking devices, such as routers, bridges, and switches 
known herein as router modules or director modules. The depicted embodiment in 

20 Figure 1 has a network traffic director system 110 in a data communication system 
which links a collection of networks 120 via router modules 130 and associated links 
140 and links 150. A link 140 is a data communication path for data packets to travel 
between a network 120 and a router module 130. A link 150 is a data communication 
path for data packets to travel between two router modules 130. Network traffic is 

25 made up of individual data packets which are sent between the networks 120 via the 
router modules 130. The network traffic director system 1 10 as embodied in the router 
module 130 allows more efficient utilization of data communication systems over prior 
art linking devices. The router module 130 is configured to adapt to sudden changes in 
the data communication network, such as randomly and rapidly changing service 
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demand or load rales or topology, while optimizing utilization of the data 
communication system based on numerous factors, including traffic delays, queue 
lengths, variance in individual delays, average and variance of utilized capacity of links 
and linking devices, number of lost packets, latencies, and throughput of linking 
5 devices. 

Unlike prior art methods, for the depicted embodiment, the optimization 
methods used account for the numerous factors influencing utilization of a data 
communication system by Ibrmulal.ng a penalty function (which accounts for negative 
effects, such as lost packets, increased delays, high latency, diminished throughput. 

10 among other things) or its opposite merit function (which accounts for beneficial 
effects). The merit and penalty functions are formulated for optimization and 
determination of equilibrium points given a rapidly changing, stochastic environment of 
random demands upon a data communication system, random flows through a data 
communication system and random changes in topology of a data communication 

1 ? system. 

This slochaslic approach is far different from prior art methods, which 
rely on predictable, non-stochastic environments and utilize deterministic formulations. 
The systems and methods used by the depicted embodiment of the network traffic 
director system 110 are designed to be effectively implemented in a data 
20 communication system without having extraordinary processing requirements, resource 
needs or other demands upon the overall data communication system. These and other 
advantages will become apparent in the following detailed description. In the following 
description, numerous specific details are provided such as methods and systems used 
lo collect information from the router modules 130. to determine whether conditions are 
25 satisfied, and to adjust values related to the router modules 130 accordingly. One 
skilled in the relevant art. however, will recognize that the invention can be practiced 
without one or more of the speci fic details, or with other devices and methods. In other 
instances, well known structures or operations arc not shown or described in detail to 
avoid obscuring the description of the embodiments. 
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In order to connect the networks 120 of the data communication system 
together, the router modules 130 arc connected together via communication units 210 as 
shown in Figure 2. The communication units 210 are linked together by data 
communication pathways 212. The communication units 210 in one embodiment arc 

5 Ethernet network cards and the data communication pathways 212 are Ethernet cables. 
Ethernet is a term well known in the art referring to collision sense multiple 
access/collision detection techniques as described under the IEEE 802 series 
specifications. In other embodiments of the present invention, the communication units 
2K» and pathways 212 are other types of point-to-point data communication links. 

, () Siime of the communication units 210 shown in Figure 2 are not shown 

to be connected via any data communication pathway 212. These communication units 
210 can be connected to other router modules 130 which are not shown in Figure 2. 
The number of communication units 210 shown for each individual router module 130 
is not intended to limit the depicted embodiment since some router modules have more 

15 communication units 210 than shown in Figure 2 if necessary to link additional router 
modules. 

Each router module 130a-d. as shown in Figure 2. is connected to a data 
network I20a-d. However, the number of data networks 120 connected to any one 
router module 130 varies, and somerouter modules 130 are only used to interconnect 

20 other router modules and are not directly connected to any network 120. In other 
instances, router modules 130 can be directly connected to a plurality of networks 120. 
Each router module 130 has a neighborhood supervisor unit 214 that includes a values 
exchanger, a dynamic load balancer unit 216, a routing table unit 218 with a plurality of 
routing lahles. and a dynamic data How splitter unit 220. 

25 The neighborhood supervisor unit 214 of a home router module 130 

exchanges information with router modules that are neighbors to the home router 
module. These neighboring router modules 130 are directly connected to the home 
router module. For instance, in Figure 2 the neighbors of home router module 1 30a 
include router modules 130b. 130c, and I30d because they are directly connected to 

30 router module 130a via communication pathways 212 from one communication unit 
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210 of a neighboring router module 130b-d lo a communication unit 210 of the home 

router module 130a. 

The neighborhood supervisor unit 214 also monitors for malfunctions 
and link failures in the data communication system. Along with monitoring, the 
5 neighborhood supervisor unit 214 updates data in the dynamic load balancer 216 of the 
home router module 1 30 to be consistent with the current physical state of the data 
communication system. The neighborhood supervisor unit 214 also reports detected 
network failures to a central fault reporting device located in the data communication 
system through a central fault monitoring function. 
j0 The dynamic load balancer unit 216 uses information collected by the 

neighborhood supervisor unit 214 of the home router module 130 from the neighboring 
router modules 130. The dynamic load balancer unit 216 adjusts the routing tables of 
the routing table unit 218 based upon the information collected in order to optimize 
overall utilization of the data communication system served by the network traffic 
1 5 director system 1 10. The methods used by the dynamic load balancer unit 216 will be 
discussed further below. The routing table unit 218 provides information to the 
dynamic data How splitter unit 220 of the home router module 130 in order that the data 
packets received by the communication unit 210 of the home router module 130 are 
directed to destination networks 120 or router modules 130 in a manner that preserves 
20 the integrity of the mathematical models used by the dynamic load balancer unit 216. 

The dynamic data flow splitter unit 220 performs the actual routing 
operation of data packets received by the home router module 130 to direct the received 
data packets to other networks 1 20 or router modules. The division of labor between 
the routing table unit 218 and the dynamic flow splitter unit 220 can be varied. In one 
25 embodiment, the routing table unit 218 prepares the routing data in a suitable form for 
the dynamic data flow splitter unit 220. The routing table unit 218 then stores this 
routing data until the dynamic data flow splitter unit 220 requires the data. The 
dynamic data flow splitter unit 220 then looks up the routing data in the routing table 
unit 218 to determine how a particular data packet is to be routed. The dynamic data 
30 flow splitter unit 220 in various embodiments of the present invention use different 
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methods to determine how a received data packet should be routed. Depending on 
which method is used, a received data packet will be sent by cither the communication 
unit 210 or the dynamic data flow splitter 220 of the router module 130 to different 
router modules 130 through one of the communication units 210 or to a directly linked 
5 network 120. Four different methods of determination, also called splitting protocols, 
are typically chosen. 

Markovian splitting is the simplest of these routing protocols to 
implement in software. In the Markovian protocol, the routing table unit 218 
transforms the desired flows into equivalent routing fractions. Only the flows out of the 

10 router module 130 are relevant. For each destination, if/- is the desired flow to the 
neighhor / for all the neighbors / of the router module 130. then the fraction of the How- 
to this neighbor is />, iJ'i ' ln turn * l ' ie dynamic data ^ ow sph lter un * 1 220 
views the fraction as a probability of a Markov process. When routing a data packet, 
the dynamic data flow splitter unit 220 looks up the probabilities corresponding to the 

15 packet destination, p\, P2 Pk- The flow splitter then "casts a random die," with an 

associated probability p t of the outcome /, and sends the data packet to the lucky 
winner. 

The easiest way to cast the random die is to generate a uniformly random 
number between zero and one and determine where this number falls in the cumulative 

20 probability sequence { } given by P n ~ 0, l\ = P t .\ + /v The random numbers can be 
generated "on the fly" or they can be stored a priori. 

The second routing protocol, called a "Routing Wheel," reduces the 
randomness of the Markovian protocol and is easy to implement in hardware. Given 
the desired probabilities /?/. /??. . . . for flows to 1 through K neighboring router 

25 modules 130 of a given router module, the routing table unit 218 of the given router 
module generates N (e.g.* 100) labels. The labels identify individual router modules 
130 and are assigned among the neighboring router modules 130 according to the 
probability associated with each neighboring router module. For instance, the 
neighboring router module 130 "f" having probability p, is assigned Np f number of 

30 labels identifying that particular neighboring router module. The routing table unit 218. 
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then arranges the labels into a conceptual distribution resembling a physical circle 
where all label types are distributed fairly evenly around the conceptual circle. The 
routing table unit 218 then sets a conceptual pointer to a location on the circle. When 
the dynamic data flow splitter unit 218 services a data packet, it looks up the currently 
5 selected label on the circle. The dynamic data flow splitter unit 220 then sends the 
packet to the neighboring router module 130 indicated by the current label and then 
advances the pointer to the next label on the circle. The process never runs out of labels 
since when the pointer is at the label in position A', the pointer next advances to the 
label in position 1 . In further embodiments, a routing wheel is generated for each 
10 ongiiiaiion'dcsiination pair and quality of service level combination (OD/OoS). 
CJ/)-(><»N combinations are discussed in detail below. 

Some higher level IP protocols are designed in such a way that makes 
the arrival of the data packets in an orderly sequential fashion at the final destination 
desirable. These protocols can handle fallouts and out-of-sequence arrivals, but they 
15 become increasingly inefficient as the number of out-of-sequence arrivals grows. In 
order to increase the efficiency of such transfer protocols, the following modification of 
flow splitting is used. This modification is incorporated into the third routing protocol 
and works with cither Markovian splitting or Routing Wheel splitting. 

For the third routing protocol, the granularity of splitting is changed for 
20 the applications running the higher level protocols mentioned above. Such protocols 
can usually be recognized from a device number in a packet header. The dynamic flow 
splitter unit 220 is endowed with short-term memory. When one of the "suspect" 
device numbers appears in the header of a packet, the dynamic flow splitter unit 220 
remembers the origin, the destination, the device number of this packet, as well as the 
25 label of the neighboring router module 130 to which the packet was sent. All 
subsequent packets that have the same origin, destination, and device number that arrive 
while the memory persists are then sent to the same neighboring router module 130 as 
the original packet. In alternative embodiments, the short-term memory is optionally 
refreshed by the later arrivals. 
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The fourth routing protocol used with the dynamic data flow splitter unit 
220 relates to low latency and quality of service. Latency of a system is the time 
required to send and receive information. Some applications, such as telephony, video- 
telephony, or real-lime interactive games or simulations require low latency levels. The 
5 latency requirements of other applications such as transmissions of email are not as 
stringent. The ordinary method for lowering latency for urgent data packets is to give 
these packets a routing priority while in routing queues as discussed below. 

Achieving additional speedups of urgent data packets is realized by 
making the dynamic data flow splitter unit 220 send more of the urgent packets over 

10 faster links and paths. The dynamic data flow splitter unit 220 then compensates for 
this by sending more of the ordinary packets over slower links. The compensation can 
be adjusted to leave the overall performance of the routing system unchanged. In one 
embodiment, identification of the faster links by the network traffic director system 1 10 
in the data communication system is readily accomplished because a high correlation 

15 exists between the highly used links and the faster links. This condition is used to 
satisfy latency and quality of service requirements by directing the urgent packets 
toward the most frequented path choices for a given address group and by directing 
more of the less urgent packets away from the most frequented path choices. 

The router module 130 also has a quality of service unit 310. a toll gale 

20 unit 312. an encoder-decoder unit 314. and a central fault monitoring unit 316 as shown 
in Figure 3. The quality of service unit 310 is concerned primarily with latency 
regarding transmission of data packets in the data communication system. The quality 
of service unit 310 is used to establish separate queues for applications with different 
latency and bandwidth needs. The queues are then served by appropriate frequencies of 

25 service and processing rales. Also, billing rates are dependent upon the level of quality 
of service. The loll gate unit 312 collects the billing statistics used to bill customers of 
the data communication system. These statistics primarily focus on billing. However, 
they can be used to determine bandwidth demands and optimize routing network 
topology. 
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To ensure privacy of communication, the data packets sent across 
network links that are vulnerable to eavesdropping are encoded for transmission and 
decoded at the destination router module. The encoder-decoder unit 314 performs the 
encoding and decoding operations. This link security is not directly related to the 
5 routing operations of the router module 130. Sufficient bandwidth for the encoder- 
decoder unit 314 is required so that transmission speed of the overall router module 130 
is not impeded. 

As described above, the dynamic load balancer unit 216 uses 
information from the neighborhood supervisor unit 214 to determine parameters that the 
10 routing table unit 218 then uses to prepare routing table data. A goal of the dynamic 
load balancer unit 216 is to optimize traffic Hows in the data communication system of 
which the network traffic director system 110 is part. A method used by the dynamic 
load balancer 216 of the depicted embodiment used to accomplish this goal is described 
as follows. 

j 5 As discussed above, each link 140 between one router module 130 and 

one network 120 and each link 150 between two router modules 130 is a data 
communication path that allows for two-way traffic of data packets. Thus, each of links 
140 and 150 of figure 1 allows data packet traffic to travel in two directions either to or 
from a particular network 120 or router module 1 30. 

20 To assist in describing the systems and methods below, each direction 

that a packet can travel along a link 140 or 150 is referred to as an "arc." The term 
• arc" does not refer to any particular physical part of a link 140 or 150 because the links 
allow packets to travel in either one of two directions. Rather, the term "arc" is 
conceptual since it refers to only one direction of travel in a link 140 or 150. Since 

25 links 140 and 150 each allow for two directions that packets can travel, each link 140 
and 150 is assigned two conceptual arcs, one arc for each of the two directions of travel 
allowed for within each link. Thus, each line of Figure 1 which represents a link 140 or 
150 could be replaced by two lines representing two arcs which conceptually indicate 
the two directions of travel that arc possible for packets to take on the link. The arcs of 
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links 140 and 150 arc illustrated in Figure 4 where each link has two arrows 
representing arcs for the two directions that packets can travel for any one link. 

The depicted embodiment is not limited to links between networks 120 
and router modules 130 but could include links between other network devices or 
5 networks of any size and topology. For example, links between routers and individual 
computers or terminals and links between large networks such as the world-wide 
Internet and a country sized corporate networks and small office local area networks are 
included in the depicted embodiment. 

Within router modules 130 there is an In node 420 and an Out node 430 

10 as shown in Figure 4. Flow of the internal arcs 440 originate from the In nodes 420 and 
terminate at the Out nodes 430 of each router module 130. The internal arcs 440 allow 
for additional parameters to be used in optimizing the data communication system. To 
furnish additional parameters, each In node 420 and Out node 430 is treated similarly as 
how the router modules 130 are treated in the various formulations for the depicted 

I 5 embodiments discussed below. 

for typical flow conditions in a data communication system, an overall 
flow in a particular arc typically is a conglomeration of one or more separate flows. 
Lach separate flow is characterized by its particular combination of flow origination 
and flow destination expressed as its origination/destination (OD) pair. The origination 

20 and destination are typically expressed at the network 120 or router module 130 level, 
so it is typically not expressed at a level specifying particular devices attached to a 
network. In some embodiments, the separate flows are characterized only by OD pairs, 
so for these embodiments, the separate flows for a particular arc have unique OD pairs. 
Summing over these unique OD pairs gives the overall flow /„ v for these arcs. 

25 In the depicted embodiment, the separate flows are also characterized by 

different quality of service (OoS) levels. In this embodiment, at least occasionally, a 
particular arc will have more than one separate flow having the same OD pair, but 
having different OoS levels. To determine the overall flow/,,, for these types of arcs, 
the separate flows would be summed over both the OD pairs and QoS levels, giving 

30 unique OD/OoS combinations. Characterization of the separate flows of an arc is not 
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limited in the present invention to only OD/OoS combinations, but also incorporates 
other factors associated with traffic flow in a data communication system. For instance, 
in another embodiment, characterization of separate flows in an arc includes a device 
number of an origination or destination device attached to a network 120. 

5 The overall flow of an arc is also constrained by an upper bound and a 

lower bound. The upper bound is typically proportional to the processing capacity of 
the arc for the depicted embodiment {e.g., 0.7 or 0.8 of the processing capacity of the 
arc). Kach individual flow can also be constrained by its individual upper bound and 
individual lower bound. Again, the most common lower bound is zero for the depicted 

10 embodiment. 

The method lo optimize traffic How used by the dynamic load balancer 
unit 216 of the depicted embodiment defines an ideal theoretical stale for data packet 
traffic in the data communication system as the case where there is zero flow 
throughout the data communication system while all demands or loads by users upon 

15 the data communication system are satisfied. A solution for data flows in the data 
communication system is sought where the solution is a point in a solution space that 
has a minimum distance to the theoretical ideal state. The solution is subject to certain 
physical constraints of a data communication system and is found through convex 
duality theory or convex optimization theory. 

20 in the depicted embodiment, the solution for data flows in a data 

communication system is uniquely formulated to optimize a variety of network 
objectives and conditions compared with prior art formulations that focus on one known 
data rate objective or condition subject to conservation of data flows. Data flows are 
conserved when the amount of flow coming into a node (typically a router module 130) 

25 equals the amount of flow leaving the node. Any data either produced or destroyed at a 
node is factored into data flow conservation. 

The solution for data flows also optimizes the following uniquely 
formulated expression £'„.„(/) involving a substantially quadratic function of data 
flows in a data communication system: 

30 
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where the subscript fc 7" indicates a particular OD/OoS combination and the subscript 
~atr indicates a particular arc "ah*\ The term f hah is for flow associated with a 
5 particular OD/QoS combination % \r and a particular arc "ah". The terms r rah and g hah 
are resistance and goal parameters, respectively, for the particular OD/OoS combination 
fc *r and arc "ah~\ The multiplication factors a and \\ are used to tailor emphasis 
between the two component expressions for Since only the relative 

magnitude of a and \S are important, we implicitly assume the quantities are scaled to 

10 satisfy u h p =- 1 . 

In the form above, the expression Z: t/ r (/) is a penalty function which 
represents undesirable influences and results affecting communication flow in a data 
communication system as have been described. Optimization of the expression 
) as J Penally function would result in minimization of £„.„(/)- If the 

15 expression E tt (l (/ ) was negated it would lake on the form of the merit function. 
Optimization of a merit function would result in a maximization of the merit function. 
Other formulations that are equivalent to optimization of the penalty and merit 
functions involve determining a point of equilibrium. As known in the art. once an 
optimization formulation is described, an equivalent formulation for a point of 

20 equilibrium is determined by using mathematical techniques known in the art. 

In a preferred embodiment, r J(th = r ah = \lc ub \ = ft,/, = 0: and a and fi 
are of the same magnitude. Then the solution of the optimization problem is the same 
as the solution of the competitive equilibrium problem using only the latter part of the 
equation, that is 



25 



ah 



0/2 O* Z /'"* 
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over the non-supply/demand arcs. 

Other embodiments also include minimization of other penalty functions 
and maximization of other merit functions. Such penalty and merit functions are related 
to the depicted embodiment in that portions of these other penally and merit functions. 
5 are at least locally approximated by the expression £ u , v (/) above. These penalty and 
merit functions all are advancements compared with the prior art, since they are used to 
optimize Hows in a stochastic environment of a data communication system. 

The expression £„.„(/) incorporates factors associated with individual 
OD/OoS combinations for each arc "uh" over all the arcs in a data communication 
10 system. F.uch individual OD/OoS combination T of an arc "ah" has a particular 
resistance parameter, /;,,„/, related lo any hindrances to (low and a goal parameter ^. t ,„ 
related to the current demand or load by users. 

For the depicted embodiment, the resistance parameter for an individual 
arc "«//>" and OD/OoS combination "f is also uniquely formulated, being proportional 
1 5 to the reciprocal of the overall data flow capacity, C ',,„/. of the individual arc "ah" and 
on-OoS combination ~f\ Thus. r ( „, oc 1 1 C . where C ral , is the maximum amount of 
data How which the individual arc "ah" is capable of supporting for OD/QoS 
combination "f \ For instance, C ah = C,. ilh and /•„,, cc \/C jah \ox all /. In 

component flows, most commonly = !>•"*• The resistance parameter is also used 

i 

20 to designate that an arc for an OD/OoS combination is not functional by setting />„,, lo a 
relatively large number. The solution to the optimization of the uniquely formulated 
expression E u „ (./ ) over all the component flows./,.,,,, of each OD/OoS combination T 
for each arc "ah~ over all ares and OD/OoS combinations of a data communication 
system results in solutions of flow ./,.„,, for each OD/QoS combination T for each arc 

25 "ah" in the data communication system. 

Methods that could be used to solve the optimization of the uniquely 
formulated expression £„,,(/) include methods that have been used to solve other 
optimization formulations such as a projected gradient method, and a hill climbing 
method. The depicted embodiment uses a unique distributed optimization method 

30 incorporated in the dynamic load balancer unit 216 for optimizing the expression 
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T,lis method is uniquely formulated for stochastic network flow that is 
uncertain, rapidly changing and random and topology that can stochastically change. 
The distributed optimization method is generally easier to implement and converges 
more quickly to a solution compared with prior art methods that have been used on 
5 other formulations. The distributed optimization method also incorporates conservation 
of Hows and unique formulations of and /;,■,„/>■ 

As part of this distributed optimization method, the depicted 
embodiment defines a set of quantities at each router module 130 that are related to 
Hows. Each quantity is referred to as a "potential." The set of potentials for a 

10 particular router module 130 has one potential for each possible OD/OoS combination 
known to the particular router module 130. Possible OD pairs are determined by the 
various combinations of a data packet origination and a data packet destination. These 
combinations would not typically include combinations where both the origination and 
destination arc the same. Typically, the data packet origination and destinations would 

IS be networks 120 and router modules 130, but could include other network devices at a 
more discrete level as discussed above, such as individual computers or terminals. 
Thus, the possible OD pairs of the OD/OoS combinations for a particular router module 
130 would typically include the OD pairs derived from all data packet originations and 
data packet destinations known to the particular router module 130. 

20 Typically networks 120 and router modules 130 are at times data packet 

originations and at other times data packet destinations. As such, the same two 
components of a first OD pair could be the components of a second OD pair by 
designating the first pair origination as the second pair destination and the first pair 
destination as the second pair origination. For small sized data communication systems, 

25 a router module 130 stores one potential for each possible OD/QoS combination for the 
entire data communication system. For a larger sized data communication system, an 
aggregation scheme is used which reduces the number of potentials stored by each 
router module 130. 

An aggregation scheme reduces the number of networks 120 and router 

30 modules 130 known to any one router module 1 30 in order to keep the set of potentials 
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stored in the router module to a manageable size. As shown in an aggregation hierarchy 
510 of Figure 5. this aggregation scheme has a physical portion 512 with physical nodes 
and a conceptual portion 514 with conceptual nodes. The physical nodes include 
networks 120 and router modules 130. The conceptual nodes each serve as labels for a 

5 particular collection made up of any number of physical nodes and/or other conceptual 
nodes. The physical nodes of the physical portion 512 are typically found in a 
particular geographical region. Figure 5 only illustrates a representative example, and 
the depicted embodiment can have other regions of various sizes and locations. 

Although Figure 5 illustrates an aggregation hierarchy in terms of 

10 geography, the depicted embodiment is not limited to only hierarchies defined in 
geographical terms. Other hierarchies defined in terms of an organizational, political, 
topical, topological, or other types of structures are also included in the present 
invention. Also, f igure 5 is not intended to limit the present invention to any particular 
manner or type of geography in defining the physical portion 512 of the aggregation 

15 hierarchy 510. The physical nodes can be grouped according to other common 
geographical regions besides the manner of using cities as regions as illustrated in 
figure 5. 

l or illustration purposes. Figure 5 shows the regions of Seattle. Portland. 
San Francisco (S.F.). Los Angeles (I. .A.). Boston, New York (N.Y.), Atlanta, and 

20 Miami having physical nodes 520-534 respectively. The Seattle region and Portland 
region represented by conceptual nodes 540 and 542 are further aggregated into the 
Northwest (N.W.) region represented by conceptual node 560. Similarly, conceptual 
nodes 544 and 546 for the S.F. and L.A. regions respectively are further aggregated into 
the southwest (S.W.) region represented by conceptual node 562. Conceptual nodes 

25 548 and 550 for the Boston and N.Y. regions respectively are aggregated into the 
Northeast (N.F ) region represented by conceptual node 564. Conceptual nodes 552 
and 554 for the Atlanta and Miami regions respectively are aggregated into the 
Southeast (S.E.) region represented by conceptual node 566. The conceptual nodes 560 
and 562 for the N.W. and S.W. regions respectively are aggregated into the West Coast 

30 region represented by conceptual node 570. The conceptual nodes 564 and 566 for the 
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N.F. and S.K. regions respectively are aggregated into the East Coast region represented 
by conceptual node 572. The conceptual nodes 570 and 572 for the West Coast and 
Cast Coast regions respectively are aggregated into the USA region represented by the 
conceptual node 574. 

5 Physical nodes of the various regions in the physical portion 512 are 

directly connected with other physical nodes (typically router modules 130) in other 
regions of the physical portion. For instance, a physical node 520c in the Seattle region 
is directly connected to a physical node 522a in the Portland region through link 580. 
The various regions are connected together via this type of direct connection between 

10 physical nodes in one region and other physical nodes (typically router modules 130) in 
another region. Not all physical nodes in one region need to be connected to another 
phssical node in another region. This is because physical nodes within the same region 
arc connected to one another either directly or through other physical nodes (typically 
router modules 130) in the same region. Subnetworks formed by the aggregation 

15 scheme need to maintain connectivity. Thus, between each pair of nodes in a 
subnetwork there must be a connection. In the aggregation scheme illustrated in Figure 
5. the physical nodes 520a-c in the area of Seattle all share a common conceptual node 
540 in the conceptual portion 514 labeled Seattle. Fach physical node 520a-c in the 
area of Seattle stores one potential for each possible OD/OoS combination based on the 

20 physical nodes 520 composed of networks 120 and router modules 130 in physical area 
512 associated with the Seattle conceptual node 540. Also, under the same aggregation 
scheme, each physical node 520a-c in the Seattle area further stores one potential for 
each possible OD/OnS combination based also on the conceptual nodes that are 
blackened in on Figure 5. The resultant OD/QoS combinations include any OD pair 

25 combination based on either physical nodes, conceptual nodes or a mixture of both 
physical and conceptual nodes as the originations and destinations of the OD pairs. The 
conceptual nodes that are blackened in on Figure 5 are Seattle conceptual node 540. 
Portland conceptual node 542, N.W. conceptual node 560. S.W. conceptual node 564, 
West Coast conceptual node 570, East Coast conceptual node 572, and USA conceptual 

30 node 574. As shown in Figure 5. the physical node 520c is directly connected to 
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physical node 522a via physical link 580 and physical node 524a via physical link 582. 
Due lo these direct connections, physical node 520c also stores potentials for OD/OoS 
combinations that incorporate one or both of physical nodes 522a and 524a as either 
origination, destination or both origination and destination. Thus, physical node 520c 
5 stores more potentials due to more OD pairs than the other physical nodes 520. In an 
alternate embodiment only physical nodes in the same region as the node in question 
along with appropriate conceptual nodes are used for possible OD pair combinations. 
Tor instance in the alternative embodiment, physical node 520c only stores potentials 
for physical nodes 520a-c in physical portion 512 and the blackened conceptual nodes. 
10 In the alternative embodiment, physical node 520c does not store potentials for any 
oilier directly connected nodes such as physical nodes 522a and 524a. 

There are general rules to designate which physical and conceptual 
nodes are used to derive OD pairs for the OD/OoS combinations for a particular 
physical node, f irst, a router module 130 (at physical node 520c in the example of 
15 Figure 5) stores potentials for each OD/OoS combination having an OD pair based on 
the phvsical nodes (520b and 520c in the example of Figure 5) with which the router 
module 130 shares a common conceptual node (540 in the example of Figure 5). 
Second, if the router module (at physical node 520c in the example of Figure 5) is 
directly connected to other physical nodes (522a and 524a in the example of Figure 5). 
20 the other physical nodes are also used to derive other OD pairs for OD/OoS 
combinations for which potentials are stored in the router module (at physical node 
520c). Finally, the router module (at physical node 520c) will store potentials for 
OD/OoS combinations for OD pairs that incorporate one or both of conceptual nodes 
that are either -lineal ancestors" of the router module (at physical node 520c) or 
25 "siblings" of the lineal ancestors. For the example of Figure 5. the conceptual nodes 
that are lineal ancestors of the physical nodes 520 (including router modules 130) are 
Seattle conceptual node 540. N.W. conceptual node 560. West Coast conceptual node 
570 and USA conceptual node 574. The conceptual nodes in the example of Figure 5 
that are siblings of the lineal ancestors are Portland conceptual node 542 (sibling of 
30 Seattle conceptual node 540). S.W. conceptual node 562 (sibling of N.W. conceptual 



BNSOOCtD <WO 0024 1 64 A 1 I > 



WO 00/24164 PCTAJS99/24145 

21 

node 560). and East Coast conceptual node 572 (sibling of West Coast conceptual node 
570). 

The method for optimizing network How in a data communication 
system used by the depicted embodiment of the dynamic load balancer unit 216 is 
5 further based on a difference function cl{f iMh .p jtab ). The difference function </(//.,,/>,/^„/,) 
is evaluated for each arc *'ub" for each OD/OoS combination tor each router module 
130 in the data communication system. When the difference function d(f jMht p iM h) is 
evaluated for a particular router module 130. that router module is designated a home 
router module. The difference function d(Jj,oh.Pj\ah) is evaluated for each arc that either 

10 originates from or terminates with the home router module. Router modules 130 that 
are physically connected to the home router module are designated neighboring router 
modules. The neighboring router modules share with the home router module 130 the 
arcs either originating or terminating with the home router module potentials. 
Potentials />, if and are at the origination and destination router modules 130 

15 respectively of arc ah for OD/OoS combination 7" Thus, the home router module 130 
and neighboring router modules 130 arc either origination or destination router 
modules, depending upon the origination and destination of the particular arcs being 
evaluated. 

For a particular arc "ah" involved in the evaluation, the difference 
20 function d(f Mih -P,.ah) evaluated for each common OD/OoS combination 7** that is 
known both to the home router module 130 and the neighboring router module 130 of 
the particular arc fc 7//>" The resultant expression for the difference function d{f jMh ,p iMh ) 
for arc and OD/OoS combination 7" is 

25 d[f jah . p Jjlh ) = (r jMh / 2)* (g tgh - / jMh ) - f jMh * (p jM - Pjj, ) 



where 
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The potentials />,.„ and pj. h are potentials for router modules 130 "a" and 
respectively for the T OD/OoS combination. Flows f k . ab where k*j are flows for the 
arc "ah" associated with OD/OoS combinations other than OD/QoS combination "/". 

The first term, (r JAlh 1 1)* fe,„ - f,.J , of the difference function 
</(/,„*• />,„/.) is a simplified version of the expression for E a . { \{f jM h) where only one arc 
"„/," and one OD/QoS combination "f is considered. The difference function 
Wi .;/■•/>/ «/•> »- s subject to the following boundary conditions: 



1 0 where 



» ,,,/. = minji/ , , »„,, - X •/*.«/■ | ' 



and where /,.„/, and /„,, are lower limits, and u, Mh and u ah are upper limits on ./,.„„ and ./„/., 
1 5 re\{>eclivcly. 

An intuitive explanation of the difference function d(fj Mb ,p JM b) is that the 
first term related to /•:,,„(//.„,,) represents undesirable results and conditions of data 
packet traffic being transmitted over a data communication system, such as delays, poor 
utilization of resources, poor adaptability to sudden changes, etc. For a given arc. when 
20 (he goal parameter, equals zero and the resistance parameter equals the inverse of 
capacity of the arc. then the first term of the difference function d(Jj.„b.Pj.ah) equals the 
number of jobs per unit time that have to go into a waiting queue because a router 
module 130 is busy. The second term fj.„ h *<J>j.a - Pj.h) represents effective utilization of 
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the data communication system as subject to constraints imposed by the data 
communication system such as flow constraints. The difference function d(Jj.ohPj.ah) 
thus represents a difference between desirable and undesirable aspects involved with 
the data communication system. 

5 As part of finding a local equilibrium point, by differentiating the 

difference function </(/;.„/,./>,.„/,) at each arc between router modules with respect to the 
How /, „/, of the OD/OoS combination ';/*' and arc "ah'\ setting the resultant differential 
equal lo zero, and solving for the flow. ./,.„/,. a response function results for flow f juh . 
As illustrated in Figure 6. for a particular arc "ah" 610 having beginning router module 

1 0 •'</" 620 and ending router module "h" 630 and for a common OD/OoS combination 7"\ 
the response function of the optimal flow f i ah 640 is: 

//.«./•(/') -"- max {'./ .«./»• min {'"'./.<i/. ■«<«/. - (/V< ~ /V l'' '/■»'>}} 

where min and max arc minimum and maximum functions which choose the minimum 
or maximum term of the pair of terms that are bracketed. For instance, max (a,h) 

15 chooses the maximum value between a and h. Thus, max (a. min (Ac)) first chooses 
the minimum between h and c and then chooses the maximum between a and the 
minimum choice between h and c. Minimum flow for ./;.„,, is typically zero. Maximum 
flow for /, ,,/, is typically a percentage of arc capacity. 

The goal parameter. and resistance parameter. r j(lh . are for the 

20 particular arc "«/>" and OD/OoS combination "/"". The potential /;,.„ is for the arc 
beginning router module "a" 620 and the potential p jM is for the arc ending router 
module "h" 630. Both potentials /?,.„ and />,.,, are associated with a common OD/OoS 
combination y. Typically, the origination and destination nodes for an OD/OoS 
combination arc not the arc beginning and arc ending router modules 620 and 630 

25 respectively. Also, the origination and destination nodes for an OD/OoS combination 
can be a combination of physical and/or conceptual nodes whereas the arc beginning 
and arc ending nodes are always physical since they are router modules 130. 

Summing these optimal flows for arc beginning "a" and arc ending "h" 
router modules 620 and 630 for arc "ah" over the OD/OoS combinations that are known 
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to the arc beginning and ending router modules results in the optimal flow for arc "aft". 
This response function of flow describes an optimized flow based on certain values for 
potentials. These potential values further satisfy a constraint based on how data packets 
are created and terminated in the data communication system. 
5 in the depicted embodiment, the potentials are independent variables and 

the flows are dependent variables. However, in other embodiments, different subsets of 
Ilovvs and potentials arc used so that some flows and potentials are dependent variables 
and other Hows and potentials are independent variables. 

The depicted embodiment of the invention is configured so that all data 
10 packet traffic originates and terminates at network terminals of network 120 or router 
modules 130. This means that all data packet traffic having a common destination, 
cither created at a particular router module 130 or arriving at the particular router 
module, must equal the data packet traffic with the common destination leaving the 
router module 130. This requirement is referred to as a conservation of Hows or a flow 
1 S constraint. When implementing the method of the depicted embodiment, the router 
modules 130 scattered throughout a data communication system frequently adjust the 
data packet routes in the data communication system to pursue the goal of maximizing 
net benefits for stochastic distributions of network loads and stochastic changes in 
topology. The adjustments are done by manipulating potential values according to the 

20 conservation of Hows. 

Adjustments are first needed when a data communication system 110 is 
initially brought into service. Adjustments are subsequently needed due to the 
continually changing demands or loads by users of the data communication system and 
changes in the topology of the data communication system due to growth of the data 
communication system or equipment failures within the existing data communication 
system. The method of the depicted embodiment incorporated into the dynamic load 
balancer units 216 of the router modules 130 is responsible for these adjustments made 

by the router modules. 

As stated above, the dynamic load balancer unit 216 uses information 
30 collected by the neighborhood supervisor unit 214. This collected information can now 



25 
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be described in terms of the response function for flow, f} ah , as discussed above. The 
discussion will be focused on a particular router module 130 designated the "home" 
router module, however, the home router module discussed is representative of all 
router modules in that all router modules also perform the steps described. 
5 In general, there are two main approaches that are used in the present 

embodiment to determine the Hows for a given home router module 130. In both 
approaches, arcs are distinguished as being either supply arcs or non-supply arcs. A 
supply arc either directly connects an origination network 120 or origination user device 
of a network for a given OD/OoS combination to the given home router module 130 or 

10 a router module between the network or user device and the home router mode where 
there is no branching of the arcs for the home router module and the network or user 
device. Non-supply arcs are also directly connected to the given home router module 
but are not directly connected to an origination node. 

For both approaches, the goal parameters for a supply arc for a given 

1 5 OD/OuS combination are set equal to the user demand or load placed on the supply arc. 
For instance, if user demand or load was 100 Mbits/see on a supply are for a given 
OD/OoS combination then the goal parameter for the same supply arc for the same 
OJVOoS combination would be set to 100 Mbiis/sec. The neighborhood supervisor unit 
214 senses what the demand or load is from any networks 120 on network dcviees.ihat 

20 are directly connected to the home router module 130. Also, for both approaches the 
resistance parameter for a given OD/OoS combination would be set to a relatively large 
number lor a supply arc. The large number means that there is a high cost incurred if 
the goal of the goal parameter (in this case 100 Mbits/sec) is not met. For a non-supply 
arc the resistance parameter for a given OD/OoS combination is set to the inverse of the 

25 arc capacity for the given OD/OoS combination. 

The two approaches to solve for flows for a given router module differ in 
how they deal with the goal parameters for non-supply arcs. The first approach solves 
for flow for one OD/OoS combination at a time whereas the second approach solves for 
flow for all OD/QoS combinations at once. For the first approach, ihe goal parameter 

30 for a non-supply arc is set to the negative of the historical flow for an OD/OoS 
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combination for the arc. The historical flow for an arc is the most recent flow amount 
recorded. Often times historical flows lend to change more slowly than demand or 
load, especially for arcs that have become a major artery for a data communication 
system. For instance, if the flow on a non-supply arc was most recently recorded as 50 
5 M bits/sec for an OD/QoS combination, then the goal parameter for that arc would be 
-50Mbits/sec. 

With the first approach, for a given home router module 130. the 
dynamic data flow splitter unit 220 determines historical flows from header information 
on data packets sent from the home router module. These historical flows of the home 

10 router module for sent packets arc stored in the dynamic data flow splitter unit 220. 
The historical Hows are represented by averages, variances and other expressions 
known in the art of statistics and mathematics. The neighborhood supervisor unit 214 
queries the dynamic data How splitter unit 220 for the" historical flows of sent data 
packets as needed for flow calculations or to send to other neighboring router modules. 

15 The historical flows of sent data packets are for arcs directly connected to the given 
home router module 1 30 handling packets sent by the home router module to networks 
or other router modules. In the depicted embodiment, the neighborhood supervisor unit 
214 of the given home router module 130 also receives updated historical flow 
information from other neighborhood supervisors for ares that are directly connected to 

20 the given home router module and carry data packets to the given home router module. 
A special case of the first approach is where all historical flows are equal to zero. This 
special case occurs at such limes as the initial start-up phase of a data communication 
system. 

The second approach to solve for flows for a given home router module 
25 130 uses computed flows instead of historical flows. Flows are determined for all 
OD/OoS combinations at once through an iterative process. For a K* 1 ' iteration, the 
overall flow for an arc is determined by summing all flows for the OD/QoS 
combinations for the arc solved during the k"' iteration. The goal parameter for the 
non-supply arcs for the A'h /"' iteration are then equal to the negative of the overall 
30 flows for the ares for the K! 1 ' iteration. 
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The techniques used to solve in this iterative fashion the flows for a 
home router module 130 for all OD/OoS combinations at once require extensive 
computing capability. Some embodiments use the second approach to solve for flows 
for all OD/OoS combinations at once. The depicted embodiment as incorporated into 

5 the methods of the described neighborhood supervisor unit 214 uses the first approach 
to solve for flows one OD/QoS combination at a lime for a given home router module 
starting with Step 710 of Figure 7. 

With the first approach, the neighborhood supervisor unit 214 of the 
home muter module 130 collects potentials and historical flows from all neighboring 

10 router modules 130 directly connected to the home router module in step 720 of the 
flowchart of Figure 7 showing the method used by the depicted embodiment. The 
collected potentials and historical flows are for common OD/OoS combinations known 
to both the home router module and at least one neighboring router module. Either the 
neighboring router modules or the home router module initiates transmission and 

15 reception of the potentials and historical flows collected from the neighboring router 
modules to the home router module. In the depicted embodiment, the neighboring 
router- modules initiate the transmission of their potentials and historical flows to the 
home router module. I lowever. in other embodiments, the home router module initiates 
the transmission. The dynamic load balancer unit 216 of the home router module 130 

20 then uses the collected potentials and historical flows along with the potentials and 
historical flows of the home router module to calculate, via the (low response function, 
the data packet flows for each OD/OoS. combination to and from the home router 
module for all arcs connected to the home router module. For each connection between 
the home router module 130 and neighboring router modules or networks 120, there are 

25 two arcs as described above. One arc has the home router module as the origination 
node and the other arc as the destination node. In other embodiments, only one-sided 
connections are possible so that for each connection there is only one arc. 

The neighborhood supervisor unit 214 then uses the historical flows and 
demand or load flows to update the goal parameters for all the non-supply and supply 

30 arcs respectively of the home router module 1 30 in step 722 of Figure 7. Typically, the 
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goal parameters lor non-supply arcs are set to the negative of the historical flow for a 
given OD/OoS combination and the goal parameters for the supply arcs arc set to the 
demand or load How on the arc. The neighborhood supervisor unit 214 also identifies 
any arcs that have zero overall flow based on their historical or demand/load flow so 
5 that any subsequent modifications of the home potentials arc done with the zero flow 
conditions in mind as explained further below. 

Initially, no OD/QoS combinations are selected by the neighborhood 
supervisor unit 214. so the outcome of decision 724 of Figure 7 regarding whether all 
OD/OoS combinations have been selected is "no." In such case, the neighborhood 
10 supervisor unit 214 selects an OD/OoS combination in step 726 that has not been 
chosen since the last time the start step 710 was performed. For all arcs either 
originating from or terminating into the home router module, the neighborhood 
supervisor unit 214 of the home router module passes, in step 728. all the potentials of 
the home router module and those collected from neighboring router modules for the 
15 selected OD/OoS combination to the dynamic load balancer unit 216. The 
neighborhood supervisor unit 214 also passes all the goal parameters and resistance 
parameters for the arcs of the particular router module 130 based on the most recent 
historical and demand/load flows collected by the neighborhood supervisor unit 214 to 
the dynamic load balancer unit 216. 
20 T he dynamic load balancer unit 216 then determines, for the selected 

OD/OoS combination, the Hows across the ares of the home router module. The 
dynamic load balancer unit 216 then determines, in decision step 730 of Figure 7 
whether the determined flows are conserved for the selected OD/OoS combination or 
whether the time period for iteration has expired. Flows are conserved when the 
25 incoming flows plus flows originated by the home router module equals the flows 
leaving the home router module. If flows arc not conserved for the selected OD/QoS 
combination or the time for iteration has not expired, the decision step in 730 is "no" 
and the method branches to step 732 where the dynamic load balancer unit 2 1 6 adjusts 
the potentials for the home router module 130 for the selected OD/OoS combination. 
30 After the home potentials are adjusted in step 732. flow conservation is again checked 
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in decision step 730. The home potentials are iterative!)' adjusted through the 
combination or potential adjustment in step 732 and now conservation verification in 
decision 730 until flow is conserved and decision step 730 is "yes" and the method 
branches back to step 722 to update the goal parameters. Iteration techniques to allow 

5 for rapid convergence to solutions involving iteration arc known in the art and include 
over or u nder-rclaxation techniques. In one embodiment, the potentials are adjusted 
one ai a time by the iteration process for a given OD/OoS combination until How is 
conserved. In another embodiment all the potentials for a router module for all the 
OD/OoS combinations associated with the router module are adjusted for each iteration 

10 of a Gaussian or other second, order method of determining solutions by block 
techniques known in the art. 

If the neighborhood supervisor unit 214 has identified any zero flow ares 
that have zero overall flow when the goal parameters are initially updated in step 722 
immediately alter the historical and demand Hows are obtained in step 720. then during 

15 the home potential adjustment in step 732. a goal parameter associated with a zero. How 
arc can be set to the negative of any calculated (low for the zero flow arc resulting from 
adjustments of potentials for an OD/QoS combination to ensure that the flow for the 
zero flow arc remains small. The new value for the goal parameter for the zero flow arc 
is then used to update the goal parameters in step 722. At the completion of adjusting 

20 potentials for all OD/OoS combinations, the updated goal parameter for a zero How arc 
will then be the summation of Hows over all OD/OoS combinations for the particular 
zero How arc due to adjustment of potentials. 

Once any goal parameters are updated in step 722 another OD/OoS 
combination is selected which had not been previously selected since the last 

25 performance of start step 710 and steps 728-732 are again performed for the newly 
selected OD/OoS combination. After steps 728-732 are completed, again any changed 
goal parameters are updated. .The process similarly cycles through the OD/OoS 
combinations until decision step 724 determines that all OD/OoS combination have 
been selected, whereby the 4k yes" branch is taken and decision step 734 determines if 

30 any potentials of the home router module 1 30 had been adjusted by step 732. 
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If any home potentials were adjusted, then decision step 734 branches to 
step 736 of Figure 7 where the neighborhood supervisor unit 214 of the home router 
module 1 30 reports the adjusted home potentials and historical flows to the neighboring 
router modules. Decision step 738 then determines if the period of time since the 

5 neighborhood supervisor unit 214 of the home router module 130 last reported home 
potentials to neighboring router modules is greater than a predetermined amount of 
time, 7V If the period is not greater then 7>>, decision step 738 branches back to the 
start step 710. If the period is greater than 7>. decision step 738 branches to decision 
step 740, the same step to which decision step 734 branches if no home potentials were 

10 adjusted. In decision step 740 a determination is made as to whether any adjusted home 
potentials have changed more than a predetermined threshold 7> since the last time 
electronic tables of the dynamic load balancer unit 216 storing the home potentials have 
been updated with adjusted home potentials. If any changes in the adjusted home 
potentials are greater than 7>, decision step 740 branches to step 742 wherein the 

15 electronic tables are updated with the latest values for the home potentials. The 
dynamic load balancer unit 216 subsequently updates the routing table unit 218. If 
there are no changes in the adjusted home potentials greater than 7-/, decision step 740 
branches to the start step 710. 

The present invention is not limited to the particular routines used by the 

20 depicted embodiment to report updated potentials to neighboring router modules 130 or 
in updating electronic tables storing home potentials. Other embodiments use other 
methods, intervals, or thresholds to determine when to report to neighboring router 
modules 1 30 and when to update electronic tables storing home potentials. 

In an alternative embodiment, the resistance parameter for each 

25 particular arc is a function of the flows of each OD/OoS combination. For instance, for 
a particular OD/OoS combination, the resistance, r. is as follows: 

r = -£r(//U-/)) 

where / is the flow for an arc for an OD/OoS combination and c is the capacity or 
residual capacity of the arc. It follows that since the response function expresses the 
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dependency of flow for an OD/QoS combination for an arc, the resistance parameters 
are also expressible in terms of potentials. 

From the foregoing it will be appreciated that, although specific 
embodiments of the invention have been described herein for purposes of illustration, 
5 various modifications may be made without deviating from the spirit and scope of the 
invention. Accordingly, the invention is not limited except as by the appended claims. 
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CLAIMS 

1 . In a data communication system having network traffic with randomly 
distributed demands and flows, a network traffic director system comprising: 

router modules configured to direct data packets in the data communication 
system, each router module being a home router module comprising: 

a neighborhood supervisor configured to send home potentials of the 
home router module to neighborhood supervisors of neighboring router modules and to 
receive neighbor potentials of the neighboring router modules from neighborhood supervisors 

of neighboring router modules; 

a dynamic load balancer configured to determine flows based on the 
home and neighbor potentials, to adjust the home potentials if first conditions including How 
conditions are not met. and to update routing tables if second conditions based on the 
adjusted home potentials are met: and 

a dvnamie data flow splitter configured to receive data packets from 
networks and router modules, to select a portion of the data communication system for each 
data packet 'received based on the updated routing tables wherein each received data packet is 
transmitted to the portion of the data communication system selected by the dynamic data 
flow splitter for the received data packet. 

2. The network traffic director system of claim 1 wherein each of the 
home potentials of each home router module is associated with a pair of nodes of the data 
communication system comprising an origination node and a destination node. 

3. The network traffic director system of claim 1 wherein each of the 
home potentials of each home router module is associated with a quality of service level. 

4. The network traffic director system of claim 1 wherein the 
neighborhood supervisor is configured to send the home potentials to neighborhood 
supervisors of neighboring router modules when the flow conditions are not met. 
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5. The network traffic director system of claim 1 wherein the flow 
conditions comprise conservation of flows. 

6. The network traffic director system of claim 1 wherein the dynamic 
load balancer is configured to determine Hows by using a response function associated with 
optimizing at least one of a penalty function or a merit function for stochastic demands and 
flows of the data communication system. 

7. The network traffic director system of claim 1 wherein the dynamic 
data flow splitter is configured to select the portion of the data communication system for 
each data packet received further based on a Markov protocol. 

8. The network traffic director system of claim 1 wherein the dynamic 
data How splitter is configured to select the portion of the data communication system for 
each data packet received further based on a Routing Wheel protocol. 

<>. The network traffic director system of claim 1 wherein the dynamic 
data How splitter is further configured to select a portion of the data communication system 
bused on previously selected portions of the data communication system when an address for 
a received data packet is a same type as an address stored in the dynamic data flow splitter. 

10. The network traffic director system of claim 1 wherein the dynamic 
data flow splitter is further configured to select of a portion of the data communication 
system based on a quality of service level associated with a data packet. 

11. In a communication system having network traffic with flows, a 
network traffic director system comprising: 

router modules, each router module being a home router module configured to 
store and adjust home potentials of the home router module, and configured to receive and 
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store neighbor potentials of neighboring router modules. the home router modules 
configured to determine ideal data flows using the home and neighbor potentials with an 
optimization of at least one of a merit function or a penalty function involving stochastic- 
changes in at least one of demands or topology in the communication system, the home router 
modules configured to receive and route network traffic based on the home and neighbor 
potentials. 

12. The network traffic director system of claim 11 wherein home router 
modules are configured to be a neighbor router module of other home router modules. 

13. The network traffic director system of claim 11 wherein the router 
modules are configured to continue adjusting home values to seek flow conservation until 
flow conservation is satisfied or until a time period has expired. 

14 The network traffic director system of claim 11 wherein the merit or 
penalt> function are at least locally approximated by a quadratic function of the flows. 

15. The network traffic director system of claim 11 wherein the neighbor 
potentials arc stored according to an aggregation scheme. 

I 6. The network traffic director system of claim 1 1 wherein the ideal data 
flows arc further based on resistance of arcs associated with the router module wherein each 
resistance of an arc is configured to be a function of arc capacity. 

1 7. The network traffic director system of claim 1 1 wherein the ideal data 
flows are further based on resistance of arcs associated with the router module wherein each 
resistance of an arc is configured to be a function of flows of the arc. 
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IX. The network traffic director system of claim 1 1 wherein the neighbor 
router modules are connected to the home router modules without any other router modules 
therebetween. 

19. The network traffic director system of claim 1 1 wherein the neighbor 
router modules are configured to be associated with the home router module through at least 
one of a political, organizational, topological, topical, or geographical relation. 

20. The network traffic director system of claim 1 1 wherein each router 
module is configured to adjust home potentials to an equilibrium point for a difference 
function associated with Hows of arcs associated with the router module, the difference 
function associated with differences between home potentials and neighbor potentials. 

21. The network traffic director system of claim II wherein each router 
module stores adjusted home values when adjustment of home values is greater than a 
threshold. 

22. The network traffic director system of claim 11 wherein each home 
router module transmits adjusted home values to its neighbor router modules based on how 
much time has elapsed since last transmitting adjusted home values. 

23. The network traffic director system of claim 11 wherein the merit 
function or penally function involves costs of congestion comprising at least one of traffic 
delays, high latency, diminished throughput, lost packets, or unresponsiveness to sudden 
changes in topology or loading. 

24. The network traffic director system of claim 1 1 wherein the neighbor 
router modules are connected to the home router modules wherein each connection has only 
one arc. 
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25. In a data communication system having network traffic, a network 

traffic director method comprising: 

storing home potentials associated with a home router module; 

receiving and storing neighbor potentials associated with neighbor router 

modules of the home router module; 

determining ideal flows based on the home and neighbor potentials and 
optimizing at least one of a merit function or a penalty function for random changes in 
demands or topology in the data communication system; 

determining conservation of (lows for the home router module; 

adjusting the home potentials to approach a state where flows are conserved if 

determined not to be conserved; 

reporting adjusted home potentials to the neighbor router modules; and 
routing network traffic based on home and neighbor values. 

26. The network traffic director method of claim 25 wherein the routing 
further includes using at least one of a Markov protocol, a Routing Wheel protocol, stored 
data packet addresses, or a level of quality of service of received communication traffic to be 
routed. 

27. The network traffic director method of claim 25 wherein the 
determining ideal Hows further involves a difference function associated with flows, home 
potentials, and neighbor potentials. 

28. The network traffic director method of claim 25 wherein the merit or 
penalty functions are at least locally approximated by a quadratic function of flows. 

29. In a data communication system having network traffic, a network 

traffic director system comprising: 

means for storing home potentials associated with a home router module; 
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means for receiving and storing neighbor potentials associated with neighbor 
router modules of the home router module; 

means for determining ideal flows based on the home and neighbor potentials 
and optimizing at least one of a merit function or a penalty function for random changes in 
demands or topology in the data communication system; 

means for determining conservation of flows for the home router module; 

means for adjusting the home potentials to approach a state where (lows are 
conserved if determined not to be conserved; 

means for reporting adjusted home potentials to the neighbor router modules; 

and 

means for routing network traffic based on home and neighbor values. 
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